Precision-Recall versus Accuracy and the Role of Large Data Sets

Practitioners of data mining and machine learning have long observed that the imbalance of classes in a data set has a negative impact on the quality of classifiers trained on that data. Numerous techniques for coping with such imbalances have been proposed, but nearly all lack any theoretical grounding. By contrast, the standard theoretical analysis of machine learning admits no dependence on the imbalance of classes at all. The basic theorems of statistical learning establish the number of examples needed to estimate the accuracy of a classifier as a function of its complexity (VC-dimension) and the confidence desired; the class imbalance does not enter these formulas anywhere. In this work, we consider the measures of classifier performance in terms of precision and recall, a measure that is widely suggested as more appropriate to the classification of imbalanced data. We observe that whenever the precision is moderately large, the worse of the precision and recall is within a small constant factor of the accuracy weighted by the class imbalance. A corollary of this observation is that the only cure for class-imbalance is a larger number of examples, a finding we also illustrate empirically. We further observe that for many applications high precision is actually needed, and hence these class-imbalance dependent measures are indeed more relevant than the accuracy.